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ABSTRACT 



This report describes the implementation and first 6 months 
of data collection of the required performance outcome measures for children 
receiving mental health services in Florida, as required iinder Florida's 1994 
Government Performance and Accountability Act. This summary is of data 
received from August 1996 to January 1997 and includes data for both serious 
emotional disturbance (SED) and at risk children at admission to services, at 
6 months, on the annual anniversary, and at discharge. Data were obtained for 
more than 6,900 children, of whom approximately 61 percent were male, 73 
percent were considered seriously emotionally disturbed, and 22 percent were 
judged to be at risk for SED. Approximately 73 percent lived with their 
biological or adoptive families and 78 percent were Medicaid eligible. 

Outcome measures included: (1) days spent in the community, (2) school 

attendance, (3) number of child arrests, (4) functional level of the child, 
and (5) family satisfaction with services. Data are to be used for monitoring 
contract compliance and to inform the state for planning and long-term 
decisions regarding services, consumers, and the system of care. Evaluation 
suggests that there has been widespread acceptance of the new measures and 
procedures, although insurance of continued agency compliance remains a 
problem as does scheduling and implementation deadlines. (DB) 
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Introduction 

In Florida, as in other states, there has been an increased 
emphasis on accountability and performance-based budgeting as 
guiding tenets of social policy. This has occurred, in part, as state 
governments have perceived negative public attitudes toward 
government spending. Based partly on this concern, the Florida 
legislature passed the 1994 Government Performance and 
Accountability Act (Ch. 216.0166. F.S., 1995) that requires state 
agencies to establish a performance-based budgeting process that 
will hold both the state agency and their contracted service 
providers accountable for individual consumer outcomes. 

Pursuant to this mandate, beginning July 1, 1996, the Mental 
Health and Substance Abuse program offices of the Florida 
Department of Children and Families (DCF) were required to 
include performance measures in all provider contracts. The 
state agency, in turn, contracted with the Louis de la Parte 
Florida Mental Health Institute (FMHI) to assist in the data 
collecting, analysis, and reporting process. This report describes 
the implementation and first six months of data collection of the 
performance outcome measures for children receiving mental 
health services in Florida. Some of the strengths and weaknesses 
of the approach are described as well as important considerations in 
development of state based outcome development efforts. 
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Service Context 

One of the outstanding features of the performance outcomes 
project was the very tight deadline for the development and 
implementation of procedures for the collection of outcome 
information for all children receiving services from the state. 
Beginning with little background or history of outcomes assess- 
ment, the state identified several critical questions that were to be 
addressed. Among other questions, the most important included: 

• What were the populations of interest in performance outcome 
monitoring? 

• What information, such as population characteristics, needed 
to be identified? 

• What questions needed to be answered to assess the perfor- 
mance of service providers for these populations? 

• What measures would be the most appropriate to answer these 
questions? 

• What sources of information might already exist which might 
fulfill data needs? 

• What procedures would be put in place to collect data, and 
how might these procedures be made a routine part of con- 
tracting with the state? and 

• How would the reliability and validity of the process be 
assured? 



In order to address the questions identified by the state, focus 
groups were held with consumers, family members, providers, 
advocacy groups, and staff from the Department of Education, the 
Department of Juvenile Justice, Children’s Medical Services, and 
Alcohol, Drug Abuse, and Mental Health. These focus groups 
developed quickly and had little opportunity for careful analysis, 
but were nevertheless able to identify and define two target 
populations for children and youth. These two populations 
include children with serious emotional disturbance (SED) and 
children at risk of serious emotional disturbance. 



In accordance with the federal definition, children with SED 
were defined as age 0-17 with a diagnosed mental, emotional or 
behavioral disorder of sufficient duration to meet diagnostic 
criteria specified within DSM-IV, and that resulted in functional 
limitations that substantially interfere with or limit the child’s role 
or functioning in a family, school, or in community activities. The 
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disability must also be expected to continue for a 
year and not be a temporary response to a stressful 
situation. Children "at risk" were defined as between 
0-17 years of age, with a need for mental health care 
for serious situational problems. The child must also 
be considered at high risk of developing serious 
limitations in functioning due to emotional distress 
if services are not provided. 

Outcome measures identified as being of 
greatest interest included: 1) days spent in the 
community, 2) school attendance, 3) number of child 
arrests, 4) the functional level of the child, and 
5) family satisfaction with services. Information about 
these outcomes was to be obtained from both pub- 
lished and public domain instruments. Demographic 
data and information pertaining to arrests, time 
spent in the community, and school attendance, 
were collected from the newly created Children’s 
Mental Health Scoring Sheet (CMHOSS). The 
functional level of the child was measured by the 
Global Assessment Scale for Children (C-GAS: 
Shaffer, et al., 1983) and supported by information 
from the Child and Adolescent Functional Assess- 
ment Scale (CAFAS: Hodges, 1996). Family 
satisfaction with services was defined as family 
centered behaviors as assessed by the Family Centered 
Behavior Scale (FCBS: Allen, Petr & Brown, 1995). 

Method 

Beginning in August, 1995, the state initiated the 
collection of performance outcome data for all 
children receiving services funded by the Florida 
Department of Children and Families or by the 
federal government through Medicaid. Training for 
all service agencies was conducted through regional 
meetings and workshops held within the state. Data 
were to be collected for all children beginning with 
the child’s admission to a service provider and 
again at six months, twelve months, and if still 
present, on the annual anniversary of the child’s 
admission. The final set of data was scheduled for 



collection at the time of the child’s discharge. Data 
collection was integrated as a regular part of the 
responsibilities of clinical staff of the service provider. 
Provider staff were assigned the responsibility of 
completing the CAFAS, the CMHOSS- which 
included demographic information, services, 
current residential setting-and the C-GAS. The 
completed forms were submitted to FMHI for 
analysis. To ensure that all children receiving 
services in the state were included in what was to 
become the statewide database, an initial assessment 
was conducted for all children receiving services as 
of August 1, 1995 regardless of the length of treat- 
ment with the state. 

The service agency also provided the parent or 
family member a copy of the Family Centered 
Behavior Scale. The family member was encouraged 
to complete the survey and return it to the agency for 
submission or mail it directly to FMHI for analysis. 

Results 

This summary includes data received during 
the first six months of data collection, August, 1996 
to January, 1997. Data included were collected for 
both SED and at risk children at admission, at 6 
months, on the annual anniversary, and at discharge. 
During this period, data were obtained for over 
6,900 children. Approximately 61% of the children 
were male, and the age of children receiving services 
ranged from a few months of age to over 18 years, 
with approximately 74% between the ages of 5 and 
15 years old. A majority of the children (73%) lived 
with their biological or adoptive families. Other 
living arrangements included inpatient settings, 
residential treatment, group homes, foster homes, 
and emergency shelters. Approximately 78% of the 
children were Medicaid eligible. Approximately 
73% of the children were considered seriously 
emotionally disturbed, while 22% were considered 
at risk for SED. 
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Discussion 

The data collected as part of the performance 
outcomes project will serve at least two major 
functions. First, the data are to be used for monitoring 
contract compliance, as the state is interested in 
determining if provider agencies are fulfilling their 
contract obligations to the state and to the consum- 
ers in their care. To provide monitoring information, 
a report format was developed that provided 
average scores for the service provider on each of 
the performance outcomes. As seen in Table 1, the 
report format provides average scores on each 
performance indicator for the agency along with a 
state mean. The data are provided for each assessment 
interval and a separate report is provided for each 
target population. 

Additional information on this report includes a 
count of the number of consumers the agency has 
served. The intent is to provide direct and timely 
information to service providers, district staff, and 
state officials. Agencies may compare the number of 
consumers served to their negotiated contracts, and 



may compare the functioning of the consumers they 
serve as measured by the performance indicators to 
the average of all agencies. District and state officials 
can use the same information to ensure that agencies 
are serving priority populations and consumer 
groups who are most in need of services. 

The second use of data from the project is to 
inform the state for planning and long term decision 
making regarding services, consumers, and the 
system of care. For example, state planners were 
especially interested in access to performance out- 
come measures for children with SED and children at 
risk for SED. Available measures included days spent 
in the community, days spent in school, and func- 
tioning as measured by the C-GAS and the CAFAS. 
Average days in school and days in the community 
for children with SED and at risk for SED are pro- 
vided in Table 2. Visual inspection of the means 
appear to reveal few if any real differences between 
children with SED and children at risk for SED in the 
days spent in school or days spent in the community. 



Table 1 

Childrens Year-to-Date Outcome Report 
"SED" Target Population 
(Quarters I - IV, FY 1996-97) 

Performance Outcomes 



Provider Number: 999999999 Total Children Served: 304 ‘ 

, Provider Name: NAME Contracted to Serve: 300 I 





Days in Community 


School Days 


C-GAS Scores 


CAFAS Scores 


Purpose 


Prov. 

Avg 


State 

Avg. 


Z 


Exhibit 

D 


Prov. 

Avg. 


State 

Avg. Z 


Prov. 

Avg. 


State 

Avg. Z 


Prov. 

Avg. 


State 

Avg. Z 


Admission 


27 


26.29 






21.40 


21.03 


53 


55.47 




40.79 


6 mo, annual 


30 


27,80 






22.04 


21.97 


53 


59.72 


40 


33.22 


Discharge 


30 


27,22 






23.00 


21.30 


60 


61.14 




29.96 


Post-admission 


30 


27,03 


0.48 


20 


22.13 


21.48 0.38 


54 


59.22 -5.04 


40 


33.06 -0.62 
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The lack of apparent differences between the 
target populations with regard to these performance 
outcomes may be contrasted with the differences 
found between the two populations on the CAFAS. 
As seen in Table 3. large differences exist between 
the two target populations in average CAFAS 
scores. Based on this preliminary data, differences 
that exist at admission between the two populations 
also carry over to discharge. Further analysis of the 
CAFAS data suggests that the differences in these 
two populations may extend to differences in the 
kinds of problems children present at admission. 
Future research will investigate differences in target 
populations and implications for the possible 
revision or addition of new target populations. 

In addition to using the data to understand target 
groups, the data may also be used to investigate 



Table 2 

Average Days in Community and Days in School 
for Children with SED and At Risk for SED 



regional differences with regard to special populations 
and service needs and to understand the service 
system and its components, such as the use of 
residential placement. Finally, analysis of the data 
will provide important feedback regarding the value 
of the measures themselves. Preliminaiy analysis of 
regional system differences already has begun. 



SED 


At Risk 




Community 


School 


Conununity 


School 




Days 


N 


Days 


N 


Days 


N 


Days 


N 


Admission 


26.5 


3544 


19.1 


2574 


26.1 


979 


19.8 


581 


6 month/Annual 


24.1 


489 


16.5 


224 


28.1 


148 


19.8 


72 


Discharge 


24.6 


437 


18.3 


247 


25.9 


189 


19.3 


99 



Table 3 

Average CAFAS Scores for Children with SED and At Risk for SED 



SED 


At Risk 




Average 


SD 


N 


Average 


SD 


N 


Admission 


57.8 


(27.4) 


1912 


37.5 


(26) 


367 


6 month/Annual 


62.3 


(27.5) 


87 


39.5 


(34.9) 


19 


Discharge 


50.0 


(31.4) 


1655 


35.9 


(31.1) 


59 



Implications for the Field 

The first six months of the performance out- 
comes process has produced both successes and 
dilemmas. Among the successes has been the wide 
spread acceptance of new measures and procedures 
forced on numerous service providing agencies on 
short notice. The success of this implementation 
phase of the outcomes project is largely the result of 
an active process of technical assistance offered to 
local agencies by phone and face to face contact. In 
spite of the overall acceptance of 
the new protocol, the dilemma 
remains regarding how to 
ensure continued compliance for 
all agencies, and how to quickly 
and efficiently communicate 
new information and changes 
in policies and procedures to 
the field. 

Related to this dilemma are 
the problems brought about by 
a largely top-down process that 
did not actively involve the 
service providers. Channels of 
communication that might 
otherwise have been established 
early in the development of the 
project do not exist. Without 
procedures in place for clear 
communication among service 
providers, district and state staff, 
and those responsible for data 
analysis and reporting, there is 
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little formal opportunity to benefit from the experi- 
ences and insights of the project participants. 

A third dilemma relates to the scheduling and 
implementation of the project. One of the most 
difficult tasks has been ensuring that final decisions 
on policy are made before project phases are initiated. 
For example, legislatively mandated deadlines v^ere 
often put into place before final policy decisions 
were made. Our best solution for these problems 
has been to help identify clear lines of authority, 
clear lines of communication, and an emphasis on 
creative problem solving. 
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